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Abstract 

Current  and  emerging  missions,  including  active  surveillance  and  tracking,  terminal  guidance  and  search  and  rescue,  require 
image-feedback  from  camera-equipped  vehicles.  Certain  mission  scenarios  and  sensor  restrictions  may  require  the  collaboration 
of  assets  over  an  ad-hoc  network.  The  development  in  this  paper  extends  efforts  to  balance  trade-offs  between  asset/sensor  cone 
positioning  to  satisfy  mission  requirements  and  network  requirements  such  as  maintaining  network  connectivity.  To  address  the 
trade-offs  between  asset  positioning  and  network  connectivity,  a  prioritized  task-function  based  guidance  law  is  developed  for  a 
simple  scenario  containing  three  assets  tracking  teams  of  mobile  targets.  One  developed  task- function  maintains  a  communication 
network  by  ensuring  the  distance  between  the  UAV’s  does  not  exceed  a  critical  threshold.  Additional  task-functions  enable 
assets  to  keep  targets  of  interest  in  the  image  cone  by  regulating  image  features  derived  from  the  camera  view.  The  UAV’s 
are  modeled  as  rotorcraft  using  an  LTI  model  equipped  with  a  gimbaled  camera.  Early  simulation  results  are  provided  to 
examine  the  behavior  of  the  assets  for  different  configurations  of  objects  observed  by  the  asset  cameras.  Future  efforts  will 
seek  to  improve  performance  by  modifying  the  control  law,  possibly  adding  time  varying  tasks  shaped  by  trajectory  planning 
techniques. 


I.  Introduction 

Unmanned  Aerial  Vehicles  (UAV’s)  are  equipped  with  imaging  sensors  in  many  scenarios,  including  image-based  guidance, 
navigation,  and  control;  geolocation;  environment  mapping;  target  tracking  and  surveillance;  etc.  A  historical  problem  in 
image-based  estimation  and  control  literature  is  controlling  the  motion  of  the  camera  to  ensure  that  targets  of  interest  stay  in 
the  field-of-view  (FOV)  (cf.  [1]— [5]).  This  problem  is  especially  difficult  when  the  camera  is  monitoring  an  adversarial  target 
in  a  dynamic  environment,  or  when  it  is  mounted  on  a  vehicle  with  motion  constraints.  A  practical  approach  to  alleviate 
the  FOV  problem  is  to  use  multiple  imaging  sources  [6],  [7],  such  as  cooperative  camera-equipped  UAV’s. 

The  use  of  cooperative  assets  is  a  well-accepted  approach.  Current  and  emerging  scenarios  (e.g.,  wide  area  surveillance, 
environmental  monitoring,  search  and  rescue/destroy)  can  exploit  coordination  between  multiple  assets  for  improved  perfor¬ 
mance.  In  many  applications,  such  as  tactical  operations,  portions  of  the  network  linking  the  collaborative  assets  may  entail 
a  mobile  ad  hoc  network  (MANET).  Assets  operating  within  a  MANET  are  faced  with  challenges,  such  as  maintaining 
connectivity  due  to  the  changing  topology  and  link  conditions  of  the  network.  These  problems  are  exacerbated  by  high 
mobility,  which  is  characteristic  of  networks  of  UAV’s. 

Fig.  1  illustrates  an  example  scenario  of  a  network  of  UAV’s  with  cameras.  In  this  scenario,  a  network  of  UAV’s  is  tracking 
red  forces  and  blue  forces  in  an  urban  environment.  The  imaging  goals  are  challenging  because  the  urban  environment  limits 
the  FOV,  and  the  cameras  are  attached  to  UAV’s  with  motion  constraints.  Coordinating  the  UAV’s  is  challenging  because 
network  connectivity  must  be  maintained  in  the  presence  of  path  loss,  shadowing,  and/or  multi-path  fading.  Even  with  perfect 
location  information,  maintaining  a  fully  connected  network  topology  can  be  challenging.  The  potential  tactical  advantages 
of  the  scenario  in  Fig.  1  motivate  the  need  for  directed  methods  that  manage  trade-offs  between  UAV/sensor  cone  positioning 
to  satisfy  mission  requirements  and  network  requirements  to  ensure  effective  collaboration  between  the  UAV’s.  Yet,  literature 
that  focuses  on  such  issues  appears  sparse. 

This  paper  presents  an  extension  of  our  earlier  work  [8]  addressing  some  of  the  challenges  imposed  by  balancing  camera 
positioning  for  image  tasks  against  UAV  positioning  for  efficient  network  operation.  Motivated  by  the  scenario  depicted  in 
Fig.  1,  the  imaging  task  involves  a  collection  of  UAV’s  keeping  a  set  of  moving  objects  in  the  camera  FOV’s,  while  also 
positioning  the  UAV’s  to  ensure  network  connectivity.  For  example,  one  UAV  tracks  the  blue  force,  while  the  other  UAV’s 
monitor  sub-groups  of  the  red  force.  The  network  is  modeled  as  undirected,  and  successful  communication  is  assumed 
possible  if  two  nodes  are  within  some  specified  maximum-link  distance.  The  investigation  is  extended  over  our  previous 
work  by  including  realistic  motion  constraints  on  the  camera  using  a  rotorcraft  model  [9]  with  a  single  gimbaled  camera. 
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Fig.  1 .  Networked  collection  of  UAV’s  performing  image-based  red  force  and  blue  force  tracking  in  an  urban  environment. 


To  manage  imaging  and  network  trade-offs,  a  series  of  low  dimensional  task- functions  are  defined,  based  on  current  image 
features  and  the  distance  to  the  nearest  neighbor  in  the  network.  For  the  image-based  task  functions,  the  desired  task- function 
velocity  is  mapped  to  a  time-varying  feature  velocity  through  corresponding  task-function  Jacobians.  These  task-function 
Jacobians  are  underdetermined  and  are  suitable  for  task-priority  kinematic  control  [10],  [11].  The  time-varying  image  feature 
velocity  is  mapped  to  camera  motions  through  image-based  visual  servoing  (IB VS)  methods  [12].  The  resulting  controller 
allows  features  to  move  within  the  image,  and  the  camera  will  move  to  keep  the  features  in  the  FOV.  To  maintain  network 
continuity,  an  additional  task  is  introduced  to  regulate  the  position  of  each  camera  to  maintain  distance  to  its  nearest  neighbor. 
If  each  UAV  is  within  a  maximum  distance  to  its  nearest  neighbor,  the  network  of  three  nodes  will  stay  simply  connected. 

Underdetermined  task- function  based  approaches  have  been  developed  for  kinematic  control  of  vehicle  platoons  in  [13], 
[14].  Full-rank  task  functions  were  used  in  vision-based  control  in  [15].  In  contrast,  we  utilize  low-rank  (underdetermined) 
utility  functions,  which  impose  fewer  constraints  on  camera  motion.  The  initial  development  of  these  underdetermined 
functions  appeared  in  [16]. 

The  problem  of  group  coordination  of  mobile  robots  for  connectivity  maintenance  has  been  studied  in  several  papers 
(cf.  [17]-[20]).  Most  of  these  studies  are  concerned  with  the  robots  moving  toward  a  goal  destination  while  maintaining 
wireless  connectivity.  In  other  words,  the  position  of  the  mobile  agents  -  if  there  were  no  need  to  maintain  connectivity 
-  would  tend  to  certain  fixed  location  in  space  depending  on  the  motion  objective.  In  contrast,  this  paper  focuses  on  the 
situation  when  there  are  no  limiting  goal  positions.  Rather,  the  unpredictable  motion  of  the  targets  may  cause  the  UAV’s  to 
move  in  a  way  that  will  sever  connectivity  in  the  absence  of  a  connectivity  maintaining  control  law. 

Maintaining  connectivity  among  moving  agents  with  second  order  dynamics  has  been  examined  in  [21],  [22].  However, 
the  uncontrolled  motion  of  the  agents  in  [21]  was  only  due  to  initial  velocities  and  second-order  dynamics,  not  due  to  the 
need  for  maintaining  targets  within  the  FOV.  A  similar  problem  formulation  was  also  considered  in  [22]  for  first  order 
dynamic  agents.  In  contrast  to  these  studies,  this  paper  examines  the  situation  when  sensing  and  maintaining  connectivity 
are  in  conflict,  e.g.,  when  motion  of  the  targets  may  cause  the  UAV’s  to  move  away  from  one  another  while  maintaining 
connectivity  may  require  them  to  stay  close. 

II.  Rotorcraft,  Camera  and  Network  Model 

Consider  a  velocity  vector  for  a  rotorcraft  given  by 

'F/i  (t)  =  \Vxh-)  Vyh->  Vzhi  ^xhi  ^yhi  ^ zh\  C  M  (1) 

where  vx  (t),  vy  (t)  and  vz  (t)  describe  the  linear  velocity,  and  ux  (t),  c oy  (t)  and  ujz  (t)  are  the  angular  velocity,  all  measured 
in  the  body  fixed  frame  such  that  the  x-axis  is  oriented  along  the  craft  major  axis  (from  tail  to  nose),  the  y- axis  points  out 
the  right  side  of  the  rotorcraft  and  the  z- axis  is  oriented  through  the  bottom  of  the  rotorcraft  body. 

A.  Rotorcraft  Model 

The  model  used  here  was  first  developed  by  Shim,  et  al.,  [9].  It  is  a  linear  time-invariant  model  of  a  rotorcraft  in  a  near 
hover  state,  giving  six  degree  of  freedom  (DOF)  motion  of  the  rotorcraft.  The  model  is  given  by 


qh  =  Ahqh  +  Bhuh. 


(2) 


where 


Qh  U  [vxh,vyh,ujxh,ujyh^,®,a,b,vzh,ujzh,r}T  G 
u  =  [ui,u2luz,U4\T . 


DU 


(3) 

(4) 


In  (3),  T>(£),  @(t)  are  the  roll  and  pitch  angle,  r(t)  is  a  feedback  gyro  rate,  and  a(t),  b(t )  are  the  blade  flapping  angles.  In 
(4),  u\ (t)  and  u2(t)  are  inputs  to  the  flap  angles,  us(t)  is  an  input  to  the  pitch  angle,  and  u±(t)  is  an  input  to  the  yaw  rate. 


The  state  matrix  A  G  Mllx11  and  input  matrix  B  G  Mllx4 
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where  the  scalar  g  is  the  gravitational  constant.  See  [9]  for  more  details,  including  the  eigenvalues  of  the  developed  system. 

Shim,  et  al.,  develop  a  stabilizing  controller  composed  of  three  control  loops  [9].  The  first  control  loop  controls  rotorcraft 
attitude,  the  second  loop  control  the  linear  velocity,  and  the  outer  loop  controls  position.  In  contrast,  we  propose  a  multi-input 
multi-output  (MIMO)  control  strategy.  Using  LQR  design  to  solve  for  a  constant  gain  matrix  Kh,  (2)-(4)  can  be  rewritten 
as 

Qh  =  Ahqh  +  BhKhqd. 


The  input  signal  qd(t )  G  M11  is  given  by 

qh  =  [ Vxhd ,  Vyhd ,  0, 0, 0, 0, 0, 0,  vzhd,  uzhd,  0]T  (5) 

where  the  subscript  d  designates  a  desired  value  of  the  appropriate  variable.  This  enables  the  rotorcraft  to  track  a  desired 
trajectory  and  heading  in  Euclidean  space, 

B.  Camera  Model 

Consider  a  camera  with  coordinate  frame  Tc  (t)  as  shown  in  Fig.  2.  The  camera  views  a  collection  of  k  feature  points  in 
front  of  the  camera.  These  points  have  coordinates  Mi  (t)  G  M3  defined  as 

Mi  =  [XhYi,  Zi]T ,  Vie{l...fc} 

in  the  camera  frame.  An  image  of  the  points  is  captured,  resulting  in  a  projection  to  a  set  of  points  in  the  image  plane. 
These  image  points  are  given  by  the  normalized  coordinates 

m  =  ,T,1]t  =  [xi,yi,l]T,  Vi  €  {l...k}. 

^ i 

Since  the  last  element  of  the  three-dimensional  normalized  coordinates  is  superfluous,  it  will  not  be  considered  in  the  sequel. 
Define  the  coordinates  mi  (t)  G  M2  as 


rn  =  [xi,yi}T 


(6) 


with  velocity  rhi(t)  E  M2  in  the  image  plane  given  by 

m  =  [xi,yi]T . 


(7) 


•  * 


Given  the  collection  of  k  feature  points,  along  with  their  coordinates  and  velocity  vectors,  a  state  position  vector  m(t) 
and  state  velocity  vector  m{t)  are  defined  as 

T  T  TiT 

ra  f?  [tt*!  ,  m2  ,  •  •  •  ,  mk  \ 
ra  =  [ra*  ,  ra2  ,  •  •  •  ,  rafc  ]  . 


The  velocity  of  the  features  points  in  the  image  plane  is  given  as  a  function  of  the  camera  velocity  vc  ( t )  =  [vxc,  vyc ,  u;: 

M6  by  the  relationship 

rri  m  Lvc,  (8) 


where  L(t)  E  R2/ex6  is  the  image  Jacobian.  The  image  Jacobian  for  the  feature  points,  L(t)  E  M2/cx6,  is  given  by 
concatenating  a  set  of  k  submatrices  Li  (t)  E  M2x6  [15],  with  Li  ( £ )  given  as 


i:  0  |  -x^  l  +  Xi  -yi 

0  ~z~  f  Vi  xiyi  Xi 


(9) 


In  the  case  that  the  feature  points  are  not  static  in  an  inertial  frame  world  frame  (e.g.,  the  feature  points  are  tracked  on 
moving  objects),  the  time  derivative  of  the  feature  points  is  given  by 


171  =  Lvc  +  £, 


(10) 


where  s(t)  is  an  unknown,  bounded  function. 

In  this  development,  the  camera  is  attached  to  the  body  of  the  helicopter.  For  simplicity,  we  assume  the  origin  of  the 
camera-fixed  axes  corresponds  to  the  origin  of  the  rotorcraft-fixed  axes.  The  camera  has  a  single  angular  degree  of  freedom 
(such  as  provided  by  a  single  gimbal)  about  the  camera  y- axis,  and  the  camera  y- axis  and  rotorcraft  y- axis  are  colinear. 
Thus,  the  camera  has  linear  and  angular  velocity  given  by 
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where  0{t)  is  the  angle  of  the  camera  relative  to  the  rotorcraft  (e.g.  the  gimbal  angle),  and  uJoc(t)  is  the  angular  rate  of  the 
camera  (e.g.  the  gimbal  angular  rate).  The  relationships  in  (11)  and  (12)  can  be  inverted  to  give  the  controlled  rotorcraft 


,uzc,]T  £ 


Fig.  3.  Illustration  of  network  model 


velocity  Vhd(t )  as  a  function  of  a  desired  camera  velocity  vcd(t)  as 
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C.  Network  Model 


(13) 

(14) 


The  network  of  N  assets  is  modeled  as  a  graph  G  =  (V,  if)  with  iV  vertices.  Each  UAV  j,  j  G  [1, . . . ,  N]  is  a  vertex 
of  the  graph,  and  is  located  at  a  position  Tj.  There  is  an  edge  E(i,j)  between  two  assets  i  and  i  ^  j,  if  the  distance 
between  them  is  less  than  a  prespecified  positive  number  r,  i.e.,  if  dij  <  r,  where  the  distance  is  defined  as 


dij  ~  \\Ti  Tj  || , 


and  r  is  a  communication  range.  This  is  illustrated  in  Fig.  3.  This  paper  considers  only  the  case  N  =  3.  In  this  case,  the 
graph  remains  connected  if  each  vertex  has  an  adjacency  of  at  least  one,  i.e.  each  UAV  is  within  a  radius  r  to  at  least  one 
other  UAV. 


III.  Task  Function-Based  Kinematic  Control 

The  guidance  and  control  objective  of  this  paper  is  to  keep  a  set  of  feature  points  within  each  camera  FOV  while  regulating 
distance  between  rotorcrafts.  To  achieve  these  objectives,  a  set  of  task  functions  are  defined  using  image  feature  coordinates 
and  the  relative  positions  of  the  rotorcrafts.  However,  the  task  functions  may  compete  in  the  sense  that  reducing  the  error  of 
one  task  function  may  increase  the  error  of  another.  To  avoid  competition  between  task  functions,  a  task-priority  kinematic 
control  [10],  [11]  is  used. 

Fet  (j>[t)  E  Mn  denote  some  task  function  of  the  feature  point  coordinates  nrii  (t)  as 

<t>  =  /Kr  •  •  ,mk) 


with  derivative 


df_ 

drrij 


rhi  =  J(m)m, 


(15) 


where  J(m)  G  lnx2fe  is  the  task  Jacobian  matrix.  The  task  functions  in  this  paper  are  of  dimension  n  <  2. 

The  objective  is  to  drive  the  feature  points  along  a  desired  velocity  rh^{t)  such  that  (j>{t)  follows  a  desired  trajectory 
(j>d(t).  Given  the  underdetermined  structure  of  the  Jacobian  matrix,  there  are  an  infinite  number  of  solutions  to  this  problem. 
The  typical  solution  based  on  the  minimum-norm  approach  [11]  is  given  as 


rricj 


=  Jt 


4>d  ~  K<t>  -  4>d) 


=  JT (jj1 


V  -1 


4>d  ~  K<t>  -  4>d ) 


(16) 


where  A  is  a  positive  scalar  gain  constant,  and  J\m)  E  R2kxn  denotes  the  minimum-norm  general  inverse  of  J  (m).  The 
desired  feature  point  derivative  ra^(£)  is  then  mapped  to  a  camera  velocity.  As  discussed  in  Section  II-B,  the  image  Jacobian 
L(t)  maps  the  camera  velocity  vc  ( t )  to  the  feature  derivative.  In  turn,  vc(t )  is  mapped  to  Vh(t )  via  (13)  and  (14).  Given 
the  velocity  and  state  constraints  imposed  by  the  rotorcraft  model,  the  following  procedure  is  used  to  generate  control  terms 
for  the  rotorcraft  and  camera  gimbal  to  achieve  a  feature  point  velocity  as  close  as  possible  to  the  desired  velocity  rin 

Define  a  five-element  vector  of  the  camera  velocity  terms  linked  to  rotorcraft  velocity  and  the  gimbal  velocity,  vc^(t)  = 
[vXc(t),  vyc(t),  vzc(t),  u)0c(t),  u;zc{t)]T .  The  corresponding  five  column  image  Jacobian  L$(t)  is  composed  of  the  first,  second, 
third,  fourth  and  sixth  columns  of  L(t),  as  defined  in  (9).  Define  a  six-element  feed-forward  vector  of  camera  velocity  terms 
that  correspond  to  uncontrolled  rotorcraft  velocity  terms,  vcff(t)  =  [0, 0, 0,  u)xc{t),  ojyc(t),  0]T.  Based  on  (16),  the  desired 
camera  velocity  vcbd  (t)  is  designed  as 

Vc5d  =  £5" +  Lvcff )  =  {LlLs)  1  £5  -  Lvcff ),  (17) 

where  L§(t)  E  M5x2/c  denotes  the  least  squares  general  inverse  used  for  L5  (t)1.  The  gimbal  velocity  is  then  given  by 
W9c(t),  and  the  remaining  camera  velocity  terms  are  then  mapped  to  rotorcraft  velocity  Vh(t)  via  (13)  and  (14). 

A  set  of  task  functions  can  be  combined  in  various  ways.  The  simplest  approach  is  to  add  the  desired  feature  velocity 
from  each  task  function  to  create  the  total  desired  feature  point  velocity.  For  example,  consider  two  tasks  (f)a  (t)  and  (j)b  (t) 
with  resulting  desired  feature  velocities  ma  (t)  and  mb  (t).  The  camera  velocity  is  then  given  as 

Vcbd  =  £5"  (rha  +  m^b  -  Lvcff) .  (18) 

Since  the  task  function  velocities  are  undetermined,  an  optional  method  is  to  choose  one  task  as  primary,  and  project  the 
other  tasks  into  the  null  space  of  the  primary  task  derivative  [10],  [11]  as 

Vcbd  =  L£  (rha  +  (I-  JUa)rnb  -  Lvcff) 

=  L+  (jt0o  +  (J  -  Jt  Ja) jfrb  -  Lvcff)  ,  (19) 

where  Ja  (ma)  and  Jb  (mb)  are  the  task  Jacobian  matrices  with  respect  to  <j)a  (t)  and  (\)b  (t),  respectively. 

The  approach  in  (19)  will  prevent  the  individual  task  systems  from  competing  and  negating  each  other,  as  the  primary 
task  will  always  be  accomplished.  Lower  priority  control  tasks  will  be  achieved  to  the  extent  that  they  do  not  interfere 
with  higher  priority  tasks.  Tertiary,  quaternary,  etc.  tasks  can  be  prioritized  by  repeating  this  process  and  projecting  each 
subsequent  task  into  the  null  space  of  the  preceding  task  Jacobians. 

IV.  Control  Development 

In  this  section,  four  task  functions  are  presented  as  part  of  a  distributed,  task-priority  kinematic  controller.  The  objective  is 
to  keep  three  sets  of  feature  points  within  the  FOV  of  three  cameras  mounted  on  UAV’s,  while  maintaining  a  communication 
network  between  the  UAV’s.  Each  camera  is  dedicated  to  observing  a  single  set  of  feature  points. 

The  first  task  function  regulates  the  distance  to  the  nearest  camera,  which  will  maintain  a  network  connection  for  three 
cameras.  Two  task  functions  are  designed  to  regulate  the  mean  and  variance  of  feature  point  coordinates.  Regulating  the 
mean  at  the  camera  center  will  keep  the  feature  points  centered  in  the  FOV.  Regulating  the  variance  will  restrict  the  distance 
between  the  feature  points  and  keep  features  away  from  the  edge  of  the  FOV.  The  third  task  function  maximizes  motion 
perceptibility,  which  ensures  desired  image  velocities  can  be  met.  These  task  functions  are  cascaded  through  null  space 
projection  and  mapped  to  camera  velocity,  as  described  Section  III. 

Chebyshev’s  inequality  proves  that  at  least  75%  of  all  values  are  within  two  standard  deviations  of  the  mean,  and  at  least 
89%  of  values  are  within  three  standard  deviations.  For  a  normally  distributed  random  process,  these  limits  are  tighter,  such 
that  approximately  95%  of  all  values  will  be  with  two  standard  deviations,  and  99.7%  of  all  values  will  be  within  three 
standard  deviations.  Consider  a  camera  with  a  512x512  pixel  FOV  and  normally  distributed  feature  points  in  the  image  plane. 
Regulating  the  mean  of  the  feature  point  coordinates  to  the  image  center  and  the  variance  of  the  feature  point  coordinates 
to  1282  will  ensure  that  at  least  95%  of  all  points  are  in  the  FOV.  For  arbitrary  distribution  of  feature  points  (e.g.  uniformly 
distributed)  points,  regulating  the  variance  to  862  will  ensure  that  at  least  89%  of  all  points  are  in  the  FOV. 

A.  Task  Function  to  Nearest  Neighbor 

For  three  camera-equipped  assets  modeled  as  a  proximity  graph,  maintaining  network  connectivity  can  be  modeled  as 
regulating  the  distance  from  each  UAV  to  its  nearest  neighbor.  Network  connectivity  will  be  maintained  if  every  UAV 
remains  within  a  bounded  distance  r  to  at  least  one  other  UAV.  To  give  the  assets  freedom  of  motion,  it  is  desirable  that  the 
distance  regulation  function  not  be  active  until  the  distance  Sij  is  beyond  a  certain  threshold  r  <  r.  To  this  end,  we  propose 

1  Since  J  ( m )  is  underdetermined  (i.e.  more  columns  than  rows),  and  L5  (£)  is  overdetermined  (i.e.  more  rows  than  columns),  the  general  inverses  have 
different  solutions  and  care  is  taken  to  denote  them  differently.  Specifically,  f  denotes  the  minimum  norm  general  inverse  used  for  J  (t) ,  and  +  denotes 
the  least-squares  general  inverse  used  for  L5  (t). 


Fig.  4.  An  example  of  the  window  function  for  p  =  2  and  r  =  3 


the  use  of  a  smooth  task  function  inspired  by  the  p-times  smooth  bump  function  given  in  [23].  This  smooth  task  function 
is  given  by 


3 

=  £< 

2=1 


“T  f(r  —  r)pdr  if  r  >  r 

r 

Si 

-jy  /  (— r  —  r)pdr  if  r  <  —  r 


else 


where  S  =  [5y,  52,  ^3]T  is  the  coordinates  of  the  nearest  neighbor  in  the  camera  frame,  — r,  r  £  M,  —  r  <  r  define 
a  window  wherein  (j)s  =  0,  and  p  £  M  gives  the  smoothness  of  the  function  Note  that  <^(t)  e=  0  if  Si  G  [— r,r]  Vi 
Furthermore,  can  be  solved  in  closed  form,  evaluates  to  a  positive  definite  function  that  can  be  differentiated  p-times, 
and  =  ±1  for  (f)s  £  [— r,r].  For  example,  for  p  =  2 


y  jV  -  z:)2* 

Z  A 


6  2  * 


6^*  ~-)3' 


These  characteristics  make  the  function  useful  for  systems  with  high  order  dynamics.  For  the  kinematic  systems  in  this 
paper,  p  =  1  is  sufficient,  in  which  case  the  polynomial  evaluates  to 


f(t-r)dt=  -{8 i-rf 
A  Z 


which  is  the  familiar  quadratic.  This  is  illustrated  for  the  case  of  p  =  2  and  r  =  3  in  Fig.  4. 
The  derivative  of  <p§  (t)  can  be  expressed  as 


=  Js  ( Vc+£5 )  , 


(20) 


where  J§  (t)  e  Mlx5  is  a  task  function  Jacobian,  and  £  M5  is  a  disturbance  due  to  the  velocity  of  the  nearest  neighbor. 
The  Jacobian  Js  (t)  is  defined  as 

J6  =  [sgn(5)T,02], 

where  sgn(-)  is  the  vector  signum  function,  and  02  £  Mlx2  is  an  all  zero  row- vector. 

The  objective  is  to  regulate  (j>s(t)  — ►  0,  under  the  assumption  that  the  current  velocity  of  an  UAV’s  nearest  neighbor  is 
available  through  the  connected  communication  network  (i.e.,  es  is  known).  Based  on  the  task  function  time  derivative  in 
(20),  a  stabilizing  camera  velocity  vcs  can  be  designed  as 

VcS  XsJsts  ^S  i 


(21) 


where  \$  is  a  positive  scalar  gain  constant.  Combining  (20)  and  (21)  yields  the  exponentially  stable  system 


=  ->^s4>s- 


B.  Task  Function  for  Mean  of  Image  Points 

Controlling  the  mean  of  feature  point  coordinates  will  help  to  ensure  the  feature  points  are  centered  around  a  position  in 
the  image  plane.  Let  </>m(t)  E  M2  denote  a  task  function  defined  as  the  sample  mean 


1  k 

=  j:12mi  =  ' 


i=  1 


The  derivative  of  cj>m  (t)  can  be  expressed  as 


(22) 


(23) 
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where  Jm  (t)  E  R2x2k  is  a  task  function  Jacobian  defined  as 

Jm  =  ^  \h,  *  •  •  ,  h\ , 

where  /2  is  the  2  x  2  identity  matrix  and  is  repeated  fc  times,  and  L(t),  vc(t)  and  £(t)  are  introduced  in  (10). 

As  described  in  [16],  to  regulate  the  mean  to  a  set  point  fmd,  PID  control  design  can  be  used  to  developed  as  a  stabilizing 
image  velocity  rhm  as 


m„ 


Jrn  (^mfme  ^ mi  J  +  ^md 


(24) 


where  (/>me(t)  m  (t>m(t)  —  </>md ,  and  Am*,  Amc/  E  f?+  are  constant  gains.  Substituting  rhm  (t)  in  (8)  gives 

=  L+rhm. 


If  k  >  3,  L+(t)  is  underdetermined,  meaning  there  are  six  degrees  of  freedom  available  in  vc{t)  to  control  more  than  six 
terms  in  rhm  (t).  This  problem  will  be  partially  addressed  by  maximizing  perceptibility  in  Section  IV-D. 


C.  Task  Function  for  Variance  of  Image  Points 

Regulating  the  variance  of  the  feature  point  coordinates  will  regulate  the  spread  of  the  feature  points  in  the  image.  That 
is,  regulating  the  variance  will  control  how  far  the  feature  points  drift  from  the  mean  value.  To  quantify  the  objective  of 
regulating  the  variance,  a  sample  variance  task  function  fv  (t)  E  M2  is  defined  as 


4>v 


1  ( xi  —  x )2 

( Vi~y  f  \  ’ 


where  x  (t)  and  y  (t)  are  the  mean  of  all  the  x  and  y  components  of  mi  (t) ,  i  E  {1 ...  k},  respectively.  The  derivative  of 
4>v  (t)  is  given  by 

(j)v  =  Jvm  (25) 

=  Jv  (Lvc  +  e) ,  (26) 

where  L(t ),  vc(t)  and  e(t)  were  given  in  (10),  and  Jv  (t)  E  M2x2/c  is  a  task  function  Jacobian  given  by 

X\  —  X  0  X2  —  x  0 
0  yx  -  y  ’  0  y2-y 

Xk  —  x  0 

5  ’  0  yk-y 


As  described  in  [16],  regulation  of  the  variance  to  a  set  point  <j)vd  can  be  accomplished  by  using  a  PID  feedback  control  to 
design  a  stabilizing  image  velocity  mv  given  by 


mv 


—  ~4  (  ^vfve  +  A* 


J  fvedt  +  \vd  —  fv 


where  </>ve(t)  =  fv(t)  —  fvd ,  and  \vi,\vd  G  M+  are  constant  gains. 


(27) 


D.  Task  Function  for  Perceptibility  of  Image  Points 

Sharma  and  Hutchinson  presented  the  concept  of  motion  perceptibility  in  [24].  Related  to  the  concept  of  manipulability 
[25],  perceptibility  gives  a  measure  of  how  well  a  camera  can  perceive  the  motion  of  objects  in  the  FOV.  Roughly  speaking, 
if  perceptibility  is  high,  small  object  or  camera  velocities  will  result  in  notable  feature  velocities  in  the  image  plane  (e.g., 
high  optical  flow).  This  is  especially  important  if  there  are  more  than  three  feature  points,  as  the  available  feature  point 
velocities  are  constrained  due  to  an  overdetermined  image  Jacobian.  Maintaining  a  high  perceptibility  helps  to  ensure  a 
larger  span  of  available  feature  point  velocity  vectors. 

Perceptibility  is  a  scalar  function  of  the  image  Jacobian  L  (t) ,  defined  as 

wp  =  J det  (LTL)  =  n  <Ji 
v  i=  1 

where  Gi  (t)  G  M+  are  the  singular  values  of  L(t).  Maximizing  wv  (t)  is  accomplished  by  maximizing  each  oi  (t).  The 
matrix  LT  (t)L  (t)  G  M3x3  is  positive  definite  and  symmetric,  so  the  eigenvalues  of  LT  (t)L  (t)  are  equal  to  a2  (t).  The  trace 
of  a  matrix  is  equal  to  the  sum  of  its  eigenvalues.  Therefore,  the  trace  of  LT(t)L  (t)  is  related  to  the  singular  values  by 

Tr(LTL 5)  =  E  *1 

i=  1 

Increasing  the  trace  of  LT(t)L(t)  will  also  increase  the  perceptibility. 

The  trace  of  LT{t)L{t)  is  given  by 

Tr(LTL5)  =  J2  fiVi  +  (Vi  + 1)2 

i= 1  v 

+  {x-i  + 1)  +  X1  +  yf ) 

A  task  function  for  perceptibility  can  be  defined  as 

^ p  =  ~k  ’ 

E  (*?  +  yf) 

i=  1 

Since  it  is  desired  to  increase  Tr(LTL),  regulating  <f>  (t)  to  0  will  result  in  increasing  the  trace.  The  time  derivative  of 
(j)p  ( t )  is  given  by 

k 

<t>P  =  -2  <P2p  E  '  Vi] 

i=  1 

—  Jp(m)m  =  Jp(m)  (Lvc  +  e)  (28) 

where  L(t),  vv(t)  and  e(t)  were  given  in  (10),  and  Jp(m )  G  Mlx2/c  is  the  task  function  Jacobian  for  perceptibility.  The 
matrix  Jp(m )  is  undefined  only  for  the  nongeneral  case  that  Vi,  rrii  =  0. 

To  regulate  <pp  (t)  —>  0,  the  feature  point  velocity  mp  (t)  G  M.2k  is  designed  as 

mp  =  -A  pJ]p(t>p,  (29) 

where  \p  is  a  positive  scalar  gain  constant.  Combining  (28)  and  (29)  gives  the  closed-loop  derivative  of  <pp  (t)  as 

(f>p  =  —Xp(j)p  +  £p, 

where  ep(t)  is  an  unknown  disturbance  caused  by  the  velocity  of  feature  points.  Despite  the  presence  of  the  disturbance 
ep(t),  the  use  of  integral  feedback  is  not  recommended  for  the  perceptibility  term.  This  is  due  to  the  fact  that  (j)p{t)  is 
unlikely  to  ever  become  zero,  leading  to  possible  integrator  windup  and  related  stability  problems. 

E.  Cascaded  Control  Law 

As  stated  in  Section  III,  the  control  objective  of  this  paper  is  to  design  a  camera  controller  that  maintains  a  set  of 
feature  points  within  the  FOV  of  several  independent  assets.  The  controller  is  decentralized,  and  each  UAV  operates  under 
an  identical  control  law.  In  addition  to  maintaining  the  view  of  the  targets,  each  camera  must  remain  within  a  maximum 
distance  from  its  nearest  neighbor  in  order  to  maintain  a  communication  network. 

Maintaining  the  connectivity  of  the  network  is  the  primary  task.  Regulating  the  mean  to  the  image  center  is  chosen  as 
the  secondary  task  in  order  to  keep  the  feature  points  centered  in  the  FOV.  Regulating  the  variance  to  a  constant  is  chosen 
as  the  tertiary  task  to  restrict  the  distance  between  the  feature  points  and  the  image  center.  These  two  tasks  ensure  features 
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Fig.  5.  Final  3D  locations  of  the  three  cameras  and  the  three  sets  of  targets  when  inter-camera  distance  is  not  regulated. 


remain  in  the  FOV.  High  perceptibility  will  allow  these  two  tasks  to  work  more  efficiently  by  ensuring  larger  available 
feature  velocities  at  lower  camera  velocities.  For  this  reason,  increasing  perceptibility  is  chosen  as  the  quaternary  task. 

The  designed  feature  velocities  given  in  (21),  (24),  (27)  and  (29)  are  used  in  the  null-space  projection  camera  velocity 
(19)  to  give  the  overall  controller  for  each  camera  as 

Vc5d  =  VCS  +  (^1  JUs)  -^5  [^m  T-  ($■  ^rn^rn)  {f^v  T-  JyJy)  ^Ip)  T-  • 

V.  Simulation  Results 

Simulations  of  the  proposed  guidance  and  control  system  are  presented.  In  these  simulations,  three  camera-equiped  UAV’s 
observe  six  rigid,  square  objects.  Each  UAV  is  responsible  for  observing  two  of  the  square  objects,  and  the  corners  of  the 
squares  give  eight  feature  points  for  each  camera  to  track.  Each  pair  of  objects  moves  with  a  sinusoidal  linear  velocity, 
independent  of  the  other  pairs.  This  simulation  mimics  the  case  of  UAV’s  tracking  sets  of  ground  vehicles.  Each  camera  has 
a  resolution  of  512  x  512.  For  each  camera  controller,  the  mean  was  regulated  to  [256,  256] T,  i.e.,  the  mean  of  the  points 
was  regulated  to  the  image  center.  The  variance  of  the  points  was  regulated  to  [1002,  1002]T,  i.e.  a  standard  deviation  of 
100  pixels.  The  simulation  was  executed  for  20  seconds  at  30  frames  per  second. 

The  first  simulation  does  not  include  any  attempt  to  regulate  the  distance  between  the  UAV’s,  i.e.  the  UAV’s  are  free  to 
move  as  necessary  to  track  the  targets,  up  to  the  constraints  imposed  by  the  rotorcraft  model.  Fig.  5  shows  the  final  3D 
positions  of  the  three  UAV’s  and  the  three  pairs  of  targets,  and  the  trajectory  each  took  over  time.  The  UAV’s  move  away 
from  each  other  to  track  their  targets.  Fig.  6  shows  the  views  of  the  three  cameras,  including  the  final  image  of  the  tracked 
targets  and  the  trajectory  the  comer  points  traced  in  the  image  over  time.  From  the  trajectory  curves,  it  can  be  seen  that  the 
objects  remain  in  view  of  each  camera  The  dashed  ellipse  and  square  represent  the  final  values  of  the  variance  and  mean, 
while  the  solid  ellipse  and  star  represent  the  goal  variance  and  mean.  It  can  be  seen  that  the  targets  remain  in  the  field  of 
view  throughout  the  simulation. 

The  second  simulation  included  the  regulation  of  distance  between  the  rotorcrafts  as  presented  in  Section  IV- A.  Fig.  7 
shows  the  final  3D  positions  of  the  three  targets  and  the  three  pairs  of  targets  and  their  trajectories  over  time.  While  the 
targets  all  end  at  the  same  locations,  the  cameras  end  up  closer  to  each  other.  Fig.  8  shows  the  views  of  the  three  cameras, 
including  the  final  image  of  the  tracked  targets  and  the  trajectory  the  comer  points  traced  in  the  image  over  time.  From  the 


Fig.  6.  The  three  camera  views  and  image  trajectories  over  time  when  inter-camera  distance  is  not  regulated. 


trajectory  curves,  it  can  be  seen  that  regulating  the  distance  between  UAV’s  makes  the  tracking  task  harder,  and  two  of  the 
cameras  briefly  lose  sight  of  part  of  a  target.  The  targets  are  brought  back  in  to  the  field  of  view,  but  it  is  clear  that  the 
high-priority  network  maintenance  task  will  prohibit  the  tracking  task  from  working  at  peak  performance. 

Fig.  9  shows  values  of  the  task  functions  over  time  for  the  case  when  distance  between  UAV’s  is  not  regulated,  and 
Fig.  10  shows  values  of  the  task  functions  when  distance  is  regulated.  As  expected,  when  distance  is  regulated,  the  mean 
and  variance  tracking  error  grows  larger,  but  the  final  distance  between  the  cameras  is  smaller. 

VI.  Conclusions  and  Future  Work 

This  paper  presents  the  extension  of  a  previous  work  in  balancing  two  competing  surveillance  tasks,  using  multiple 
UAV’s  to  monitor  multiple,  moving  targets  while  maintaining  a  network  communications  between  the  UAV’s.  This  problem 
is  visualized  as  multiple  air  vehicles  equipped  with  cameras  tracking  several  groups  of  moving  vehicles,  with  a  network 
modeled  as  a  proximity  graph.  Previous  efforts  did  not  include  a  vehicle  model  or  velocity  constraints  on  the  camera,  and  this 
work  introduces  the  use  of  rotorcrafts  with  gimbaled  cameras.  To  achieve  the  tasks,  a  series  of  prioritized,  underdetermined 
task  functions  were  developed.  Maintaining  network  connectivity  can  be  ensured  by  regulating  the  distance  of  each  UAV  to 
its  nearest  neighbor.  Targets  can  be  kept  in  field  of  view  through  regulating  mean  and  variance  of  the  targets’  features  in 
the  image.  A  third  task  function  seeks  to  maximize  motion  perceptibility.  There  is  no  specific  goal  image  or  goal  pose  for 
the  cameras,  rather  the  underdetermined  nature  of  the  task  functions  allows  the  camera  to  move  as  necessary  to  regulate  the 
task  functions  and  keep  objects  in  the  FOV  while  maintaining  network  connectivity. 

Simulations  point  towards  positive,  but  incomplete  results  at  this  time.  When  distance  between  the  crafts  is  not  enforced, 
target  tracking  shows  the  successful  extension  our  multi-target  tracking  method  to  include  realistic  vehicle  constraints. 
However,  when  the  distance  between  crafts  is  maintained,  the  targets  are  able  to  temporarily  escape,  as  enforcing  the  maximum 
distance  prevents  the  rotorcrafts  from  following  the  targets.  Future  work  will  focus  on  overcoming  this  by  improving  the 
guidance  law.  The  prioritization  scheme  may  be  too  rigid,  and  alternatives  could  be  investigated.  Another  option  is  the  use 
of  time-varying  desired  values  for  the  task  functions,  possibly  incorporating  motion  planning,  which  may  give  more  freedom 
of  movement  while  meeting  the  relative  distance  requirements. 

There  are  several  other  avenues  of  future  work.  Simulations  are  very  promising,  but  a  proper  analysis  can  determine 


Fig.  7.  Final  3D  locations  of  the  three  cameras  and  the  three  sets  of  targets  when  inter-camera  distance  is  not  regulated. 


whether,  or  under  what  conditions,  stability  can  be  achieved  for  all  tasks.  There  are  also  numerous  other  task  functions  that 
could  be  used.  For  instance,  it  may  be  desirable  to  maintain  a  certain  distance  or  orientation  with  respect  to  the  tracked 
objects.  The  network  was  limited  to  three  nodes,  which  allowed  for  the  use  of  a  simple  distance  regulation  to  maintain  the 
network.  This  methodology  must  be  extended  to  a  larger,  possibly  unlimited,  number  of  assets.  This  will  require  a  better 
metric  for  measuring  and  regulating  network  connectivity.  Finally,  we  hope  to  conduct  flight  experiments  with  the  support 
of  the  Air  Vehicles  Directorate  of  AFRL,  where  recently  an  indoor  flight  facility  for  micro  air  vehicles  has  been  developed. 
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